Development of an online service for coping with spousal loss by means of human-centered and stakeholder-inclusive design: the case of LEAVES.

To support older mourners after the loss of their partner, LEAVES, an online self-help service that delivers the LIVIA spousal bereavement intervention, was developed. It integrates an embodied conversational agent and an initial risk assessment. Based on an iterative, human-centered, and stakeholder inclusive approach, interviews with older mourners and focus groups with stakeholders were conducted to understand their perspective on grief and on using LEAVES. Subsequently, the resulting technology and service model were evaluated by means of interviews, focus groups, and an online survey. While digital literacy remains a challenge, LEAVES shows promise of being supportive to the targeted end-users.

An experiment on data sharing options designs for eHealth interventions.

Background: With eHealth technology interventions, users' personal health data can be easily shared among different stakeholders. Users should decide with whom they want to share their data. As support, most eHealth technology has data sharing options functionalities. However, there is little research on how to design these visually. In this paper, we took two possible data sharing options designs - data and party perspective - for an existing eHealth technology intervention, and we explored them. Objective: The aim was to find which of the two designs is the best in terms of trust, privacy concerns, ease of use, and information control. Additionally, to investigate how these factors influence each other with also the goal of giving practical advice on designing for privacy. Method: We conducted a between-subjects online design experiment (N = 123). After having visualised one of the two data sharing options designs, participants filled in an online questionnaire. To analyse the data, t-test analyses, correlation analyses, and backward regression analyses were conducted. Results: Information control scored higher in the data perspective condition (t (97) = 2.25, p = .03). From the different regression analyses, we found that trust and ease of use play a role in all sharing-related factors. Conclusions: We concluded that the design of data-sharing options in eHealth technology affects the experience of the user, mostly for trust and ease of use. In the end, we provided several actionable design advices on how to design for privacy.

Optimizing appreciation and persuasion of embodied conversational agents for health behavior change: A design experiment and focus group study.

Embodied Conversational Agents (ECAs) can increase user engagement and involvement and can strengthen the effect of an intervention on health outcomes that is provided via an ECA. However, evidence regarding the effectiveness of ECAs on health outcomes is still limited. In this article, we report on a study that has the goal to identify the effect of a match between a health topic and the ECAs' appearance on ratings of personality characteristics, persuasiveness and intention to use. We report on an online experiment with three different ECAs and three different health topics, conducted among 732 older adults. We triangulated the quantitative results with qualitative insights from a focus group. The results reveal that older adults prefer an ECA that has an appearance matching a certain health topic, resulting in higher ratings on persuasiveness and intention to use. Personality characteristics should be measured embedded within a health topic, but are not rated higher because of a match. We furthermore provide guidelines for designing the content of the ECA.

Developing an eMental health monitoring module for older mourners using fuzzy cognitive maps.

Objective: Effective internet interventions often combine online self-help with regular professional guidance. In the absence of regularly scheduled contact with a professional, the internet intervention should refer users to professional human care if their condition deteriorates. The current article presents a monitoring module to recommend proactively seeking offline support in an eMental health service to aid older mourners. Method: The module consists of two components: a user profile that collects relevant information about the user from the application, enabling the second component, a fuzzy cognitive map (FCM) decision-making algorithm that detects risk situations and to recommend the user to seek offline support, whenever advisable. In this article, we show how we configured the FCM with the help of eight clinical psychologists and we investigate the utility of the resulting decision tool using four fictitious scenarios. Results: The current FCM algorithm succeeds in detecting unambiguous risk situations, as well as unambiguously safe situations, but it has more difficulty classifying borderline cases correctly. Based on recommendations from the participants and an analysis of the algorithm's erroneous classifications, we propose how the current FCM algorithm can be further improved. Conclusion: The configuration of FCMs does not necessarily demand large amounts of privacy-sensitive data and their decisions are scrutable. Thus, they hold great potential for automatic decision-making algorithms in mental eHealth. Nevertheless, we conclude that there is a need for clear guidelines and best practices for developing FCMs, specifically for eMental health.

How to Prevent the Drop-Out: Understanding Why Adults Participate in Summative eHealth Evaluations.

The aim of this study was to investigate why adults participate in summative eHealth evaluations, and whether their reasons for participating affect their (non-)use of eHealth. A questionnaire was distributed among adults (aged ≥ 18 years) who participated in a summative eHealth evaluation. This questionnaire focused on participants' reason to enroll, their expectations, and on whether the study met their expectations. Answers to open-ended questions were coded by two researchers independently. With the generalized estimating equations method we tested whether there is a difference between the type of reasons in use of the eHealth service. One hundred and thirty-one adults participated (64.9% female; mean age 62.5 years (SD = 10.5)). Their reasons for participating were mainly health-related (e.g., being more active). Between two types of motivations there was a difference in the use of the eHealth service: Participants with an intellectual motivation were more likely to drop out, compared to participants with an altruistic motivation. The most prevalent expectations when joining a summative eHealth evaluation were health-related (like expecting to improve one's health). 38.6% of the participants said their expectation was fulfilled by the study. In conclusion, We encourage eHealth evaluators to learn about adults' motivation to participate in their summative evaluation, as this motivation is very likely to affect their results. Including altruistically motivated participants biases the results by their tendency to continue participating in a study.

A Web-Based Self-help Intervention for Coping With the Loss of a Partner: Protocol for Randomized Controlled Trials in 3 Countries.

BACKGROUND: The death of a partner is a critical life event in later life, which requires grief work as well as the development of a new perspective for the future. Cognitive behavioral web-based self-help interventions for coping with prolonged grief have established their efficacy in decreasing symptoms of grief, depression, and loneliness. However, no study has tested the efficacy for reducing grief after losses occurring less than 6 months ago and the role of self-tailoring of the content. OBJECTIVE: This study aims to evaluate the clinical efficacy and acceptance of a web-based self-help intervention to support the grief process of older adults who have lost their partner. It will compare the outcomes, adherence, and working alliance in a standardized format with those in a self-tailored delivery format and investigate the effects of age, time since loss, and severity of grief at baseline as predictors. Focus groups to understand user experience and a cost-effectiveness analysis will complement the study. METHODS: The study includes 3 different randomized control trials. The trial in Switzerland comprises a waitlist control group and 2 active arms consisting of 2 delivery formats, standardized and self-tailored. In the Netherlands and in Portugal, the trials follow a 2-arm design that will be, respectively, complemented with focus groups on technology acceptance and cost-effectiveness analysis. The main target group will consist of adults aged >60 years from the general population in Switzerland (n≥85), the Netherlands (n≥40), and Portugal (n≥80) who lost their partner and seek help for coping with grief symptoms, psychological distress, and adaptation problems in daily life. The trials will test the intervention's clinical efficacy for reducing grief (primary outcome) and depression symptoms and loneliness (secondary outcomes) after the intervention. Measurements will take place at baseline (week 0), after the intervention (week 10), and at follow-up (week 20). RESULTS: The trials started in March 2022 and are expected to end in December 2022 or when the needed sample size is achieved. The first results are expected by January 2023. CONCLUSIONS: The trials will provide insights into the efficacy and acceptance of a web-based self-help intervention among older adults who have recently lost a partner. Results will extend the knowledge on the role of self-tailoring, working alliance, and satisfaction in the effects of the intervention. Finally, the study will suggest adaptations to improve the acceptance of web-based self-help interventions for older mourners and explore the cost-effectiveness of this intervention. Limitations include a self-selective sample and the lack of cross-cultural comparisons. TRIAL REGISTRATION: Switzerland: ClinicalTrials.gov NCT05280041; https://clinicaltrials.gov/ct2/show/NCT05280041; Portugal: ClinicalTrials.gov NCT05156346; https://clinicaltrials.gov/ct2/show/NCT05156346. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): PRR1-10.2196/37827.

The Limitations of User-and Human-Centered Design in an eHealth Context and How to Move Beyond Them.

Human-centered design (HCD) is widely regarded as the best design approach for creating eHealth innovations that align with end users' needs, wishes, and context and has the potential to impact health care. However, critical reflections on applying HCD within the context of eHealth are lacking. Applying a critical eye to the use of HCD approaches within eHealth, we present and discuss 9 limitations that the current practices of HCD in eHealth innovation often carry. The limitations identified range from limited reach and bias to narrow contextual and temporal focus. Design teams should carefully consider if, how, and when they should involve end users and other stakeholders in the design process and how they can combine their insights with existing knowledge and design skills. Finally, we discuss how a more critical perspective on using HCD in eHealth innovation can move the field forward and offer 3 directions of inspiration to improve our design practices: value-sensitive design, citizen science, and more-than-human design. Although value-sensitive design approaches offer a solution to some of the biased or limited views of traditional HCD approaches, combining a citizen science approach with design inspiration and imagining new futures could widen our view on eHealth innovation. Finally, a more-than-human design approach will allow eHealth solutions to care for both people and the environment. These directions can be seen as starting points that invite and support the field of eHealth innovation to do better and to try and develop more inclusive, fair, and valuable eHealth innovations that will have an impact on health and care.

Time to act mature-Gearing eHealth evaluations towards technology readiness levels.

It is challenging to design a proper eHealth evaluation. In our opinion, the evaluation of eHealth should be a continuous process, wherein increasingly mature versions of the technology are put to the test. In this article, we present a model for continuous eHealth evaluation, geared towards technology maturity. Technology maturity can be determined best via Technology Readiness Levels, of which there are nine, divided into three phases: the research, development, and deployment phases. For each phase, we list and discuss applicable activities and outcomes on the end-user, clinical, and societal front. Instead of focusing on a single perspective, we recommend to blend the end-user, health and societal perspective. With this article we aim to contribute to the methodological debate on how to create the optimal eHealth evaluation design.

Overcoming Challenges to Inclusive User-based Testing of Health Information Technology with Vulnerable Older Adults: Recommendations from a Human Factors Engineering Expert Inquiry.

OBJECTIVES: Involving representative users in usability testing of health information technology (HIT) is central to user-centered design. However, (vulnerable) older adults as representative users have unique requirements. Aging processes may affect physical capabilities and cognitive skills, which can hamper testing with this demographic and may require special attention and revised protocols. This study was performed to provide expert-based recommendations for HIT user-testing with (vulnerable) older adults to support inclusive HIT design and evaluation. METHODS: First, we conducted a structured workshop with ten experts in HIT implementation and research, recruited through purposeful sampling, to generate insights into how characteristics of older adults may influence user-testing. Next, five Human Factor researchers experienced in HIT user-testing with (vulnerable) older adults validated the results and provided additional textual insights to gain consensus on the most important recommendations. A thematic analysis was performed on the resulting inquiries. Applied codes were based on the User-Centered Design framework. RESULTS: The analysis resulted in nine recommendations for user-testing of HIT with older adults, divided into three main themes: (1) empathetic approach and trust-building, (2) new requirements for testing and study design, and (3) adjustments to usability evaluation methods. For each theme a checklist of relevant items to follow-up on the recommendation is provided. CONCLUSIONS: The recommendations generated through expert inquiry contribute to more effective usability testing of HIT with older adults. This provides an important step towards improved accessibility of HIT amongst older adults through inclusive user-centered design.

Game not over: Explaining older adults' use and intention to continue using a gamified eHealth service.

BACKGROUND: Gamification within eHealth services can increase eHealth adoption. However, little is known about factors affecting adoption of gamified eHealth among older adults. In this study, we sought to explain the (continued) use of a gamified eHealth service among older adults (55+). METHODS: Participants used a gamified eHealth service, focusing on falls prevention, for 4 weeks and completed a post-test questionnaire based on the Technology Acceptance Model. We used Partial Least Squares Structural Equation Modeling to analyse our data. RESULTS: Seventy-two older adults participated with a mean age of 65.1 years (SD = 7.0). Our results show that first, perceived ease of use affected use of the service (use duration: ß = 0.303, R2 = 0.130, and use frequency: ß = 0.304, R2 = 0.107). Second, perceived usefulness affected the intention to continue using the service (ß = 0.754, R2 = 0.640). Third, use of the service did not predict the intention to continue using it. Furthermore, enjoyment affected perceived usefulness (ß = 0.783, R2 = 0.563) and aesthetics affected perceived ease of use (ß = 0.634, R2 = 0.652). CONCLUSIONS: This study refutes the expected relation between use and intention to continue use a gamified eHealth service. Additionally, we learned that using theoretical approaches focusing on technology acceptance, are not suitable for explaining (continued) use of gamified eHealth services.

The First Introduction of Social Robotics in Rehabilitation Care.

Can you imagine to receive treatment through a robot? When talking about the future of healthcare, this is the vision many people have. Currently, the predominant role of social robots in care is entertaining patients. However, this does not have an impact on care process itself. In this paper, we focus on defining use cases other than merely keeping patients' company by implementing a Pepper robot in inpatient rehabilitation setting, and expand upon usability testing the use cases. Our findings showed that, to ensure sustainable implementation of social robots in care organizations, we need excessive collaboration with the target population.

Consulting the Oracle: A Delphi study for determining parameters for a mental health user profile and personalization strategy for an online service to aid grieving older adults.

While much effort has been devoted to the development of mental e-health interventions, the tailoring of these applications to user characteristics and needs is a comparatively novel field of research. The premise of personalizing mental e-health interventions is that personalization increases user motivation and (thereby) mitigates intervention dropout and enhances clinical effectiveness. In this study, we selected user profile parameters for personalizing a mental e-health intervention for older adults who lost their spouse. We conducted a three-round Delphi study involving an international and interdisciplinary expert panel (N = 16) with two objectives. The first aim was to elicit adaptation strategies that can be used to dynamically readjust the intervention to the user's needs. The second aim was to identify a set of meaningful indicators for monitoring the user from within the grief intervention to escalate from self-help to blended care, whenever advisable. This Delphi study used as starting point an evaluated, text-based grief intervention composed of ten modules, including psychoeducation about grief and cognitive-behavioral exercises to support the user in adjusting their lives after bereavement. Every user follows this grief intervention in a linear fashion from beginning to end. The resulting conceptual adaptation model encompasses dynamic adjustments, as well as one-time adjustments performed at the initialization of the service. On the level of the application structure, the adaptations affect when which topic module is presented to the user. The adaptations further provide strategies for adjusting the text-based content of individual intervention modules dependent on user characteristics and for selecting appropriate reactions to user input. Eighteen monitoring parameters were elicited and grouped into four categories: clinical, behavioral/emotional, interactive, and external. Parameters that were perceived as most urgent to attend to for escalation were Suicidality, Self-destructive behavior, Client-initiated escalation, Unresponsiveness and (Complicated) Grief symptoms.

Use and Effect of Embodied Conversational Agents for Improving Eating Behavior and Decreasing Loneliness Among Community-Dwelling Older Adults: Randomized Controlled Trial.

BACKGROUND: Embodied conversational agents (ECAs) have been proposed as a promising interaction modality for the delivery of programs focused on promoting lifestyle changes. However, it is not understood what factors influence the health effects of ECAs or their use. OBJECTIVE: We aimed to (1) identify whether ECAs could persuade community-dwelling older adults to change their dietary behavior and whether ECA use could decrease loneliness, (2) test the pathways to these effects, and (3) understand factors influencing the use of ECAs. METHODS: A randomized controlled trial was conducted. The intervention group received access to the PACO service for 8 weeks. The waitlist group started PACO use after waiting for 4 weeks. Two primary outcomes (eating behavior and loneliness) were assessed via online questionnaires at intake, upon joining the waitlist, after 4 weeks, and after 8 weeks. The third primary outcome (use) was assessed via data logs. Secondary outcomes were measured at the same time points, via questionnaires or an optional interview. RESULTS: In total, 32 participants completed the intervention. We found a significant correlation between use in minutes on the one hand, and perceived usefulness (r=0.39, P=.03) and enjoyment on the other (r=0.38, P=.03). However, these did not predict use in the full regression model (F2,29=1.98, P=.16, R2=0.12). Additionally, PACO use did not lead to improvement in eating behavior (χ22=0.34, P=.85) or a decrease in loneliness (χ22=0.02, P=.99). CONCLUSIONS: Our study did not provide any concluding evidence about factors that are linked to the use or health effects of ECAs. Future service design could benefit from either creating a functional design catering to the predominant stage in the precaution adoption process model of the targeted population, or by personalizing the service based on an intake in which the end user's stage is determined. TRIAL REGISTRATION: ClinicalTrials.gov NCT04510883; https://clinicaltrials.gov/ct2/show/NCT04510883. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): RR2-10.2196/22186.

Improving usability benchmarking for the eHealth domain: The development of the eHealth UsaBility Benchmarking instrument (HUBBI).

BACKGROUND: Currently, most usability benchmarking tools used within the eHealth domain are based on re-classifications of old usability frameworks or generic usability surveys. This makes them outdated and not well suited for the eHealth domain. Recently, a new ontology of usability factors was developed for the eHealth domain. It consists of eight categories: Basic System Performance (BSP), Task-Technology Fit (TTF), Accessibility (ACC), Interface Design (ID), Navigation & Structure (NS), Information & Terminology (IT), Guidance & Support (GS) and Satisfaction (SAT). OBJECTIVE: The goal of this study is to develop a new usability benchmarking tool for eHealth, the eHealth UsaBility Benchmarking Instrument (HUBBI), that is based on a new ontology of usability factors for eHealth. METHODS: First, a large item pool was generated containing 66 items. Then, an online usability test was conducted, using the case study of a Dutch website for general health advice. Participants had to perform three tasks on the website, after which they completed the HUBBI. Using Partial Least Squares Structural Equation Modelling (PLS-SEM), we identified the items that assess each factor best and that, together, make up the HUBBI. RESULTS: A total of 148 persons participated. Our selection of items resulted in a shortened version of the HUBBI, containing 18 items. The category Accessibility is not included in the final version, due to the wide range of eHealth services and their heterogeneous populations. This creates a constantly different role of Accessibility, which is a problem for a uniform benchmarking tool. CONCLUSIONS: The HUBBI is a new and comprehensive usability benchmarking tool for the eHealth domain. It assesses usability on seven domains (BSP, TTF, ID, NS, IT, GS, SAT) in which a score per domain is generated. This can help eHealth developers to quickly determine which areas of the eHealth system's usability need to be optimized.

Best Practices and Lessons Learned for Action Research in eHealth Design and Implementation: Literature Review.

BACKGROUND: Action research (AR) is an established research framework to introduce change in a community following a cyclical approach and involving stakeholders as coresearchers in the process. In recent years, it has also been used for eHealth development. However, little is known about the best practices and lessons learned from using AR for eHealth development. OBJECTIVE: This literature review aims to provide more knowledge on the best practices and lessons learned from eHealth AR studies. Additionally, an overview of the context in which AR eHealth studies take place is given. METHODS: A semisystematic review of 44 papers reporting on 40 different AR projects was conducted to identify the best practices and lessons learned in the research studies while accounting for the particular contextual setting and used AR approach. RESULTS: Recommendations include paying attention to the training of stakeholders' academic skills, as well as the various roles and tasks of action researchers. The studies also highlight the need for constant reflection and accessible dissemination suiting the target group. CONCLUSIONS: This literature review identified room for improvements regarding communicating and specifying the particular AR definition and applied approach.

International eHealth ecosystems and the quest for the winning value proposition: findings from a survey study.

BACKGROUND: eHealth ecosystems are becoming increasingly important for national and international healthcare. In such ecosystems, different actors are connected and work together to create mutual value. However, it is important to be aware of the goals that each actor pursues within the ecosystem. METHOD: This study describes the outcomes of a workshop (30 participants) and two surveys (completed by 54 and 100 participants), which investigated how different types of industry stakeholders, namely social services, healthcare, technology developers and researchers, rated potential value propositions for an eHealth ecosystem. Both the feasibility and the importance of each proposition was taken into account. RESULTS: Interoperability between services was highly valued across industry types but there were also vast differences concerning other propositions. CONCLUSION: Jointly reflecting on the different perceived values of an ehealth ecosystem can help actors working together to form an ecosystem.

Contextual Health Information Behavior in the Daily Lives of People with Type 2 Diabetes: A Diary Study in Scotland.

Changes in lifestyle can have positive effects on treating type 2 diabetes (T2D), like sporting or healthy eating. Therefore, a person diagnosed with T2D is often advised to make healthy choices throughout the day, in addition to other interventions such as medication. To do this, he or she needs health information to support decision-making. Literature describes ample categorizations of types of (health) information behavior and theoretical models that explain the factors that drive people to search for, encounter or avoid information. However, there are few longitudinal studies about triggers and factors in daily life that affect health information behavior (HIB). This study was set up to identify triggers, actions and outcomes for active, passive and avoidant HIB situations in daily life among Scots with Type 2 diabetes (T2D) to identify points of attention for communication strategies. Twelve participants took part in a four-week diary study. Every day, participants received an online diary form to describe active, passive or avoidant HIB situations. Data collection resulted in 53 active, 120 passive and 25 avoidant diary entries. Seven active HIB contexts (e.g., experiencing symptoms, cooking dinner, sports training) and five passive HIB contexts (e.g., home, work, medical facility) were identified. Four motivations for avoidance were found (e.g., time constraints, no health trigger). These results can be used to supplement the theoretical models of health information behavior. Furthermore, health professionals can use these results to support their clients with T2D in the self-management of their health, by guiding them to trustworthy sources of health information and lowering barriers for searching health information.

Predictors to Use Mobile Apps for Monitoring COVID-19 Symptoms and Contact Tracing: Survey Among Dutch Citizens.

BACKGROUND: eHealth apps have been recognized as a valuable tool to reduce COVID-19's effective reproduction number. The factors that determine the acceptance of COVID-19 apps remain unknown. The exception here is privacy. OBJECTIVE: The aim of this article was to identify antecedents of acceptance of (1) a mobile app for COVID-19 symptom recognition and monitoring and (2) a mobile app for contact tracing, both by means of an online survey among Dutch citizens. METHODS: Next to the demographics, the online survey contained questions focusing on perceived health, fear of COVID-19, and intention to use. We used snowball sampling via posts on social media and personal connections. To identify antecedents of the model for acceptance of the 2 mobile apps, we conducted multiple linear regression analyses. RESULTS: In total, 238 Dutch adults completed the survey; 59.2% (n=141) of the responders were female and the average age was 45.6 years (SD 17.4 years). For the symptom app, the final model included the predictors age, attitude toward technology, and fear of COVID-19. The model had an r2 of 0.141. The final model for the tracing app included the same predictors and had an r2 of 0.156. The main reason to use both mobile apps was to control the spread of the COVID-19 virus. Concerns about privacy was mentioned as the main reason to not use the mobile apps. CONCLUSIONS: Age, attitude toward technology, and fear of COVID-19 are important predictors of the acceptance of COVID-19 mobile apps for symptom recognition and monitoring and for contact tracing. These predictors should be taken into account during the development and implementation of these mobile apps to secure acceptance.

Supporting eating behaviour of community-dwelling older adults: co-design of an embodied conversational agent.

In order to support community-dwelling older adults with healthy eating behaviours, Embodied Conversational Agents (ECAs) may be an effective and engaging medium. However, ECAs have not yet been found to be capable of engendering behaviour change, which is partly attributed to the absence of a match with users' practices, needs and preferences. Hence, we describe a co-design process with older adults that informs both the content and the appearance of an ECA. Data was gathered through three consecutive iterations of co-design sessions with two groups of community-dwelling older adults in the Netherlands. Prior to the first session, participants completed a seven-day lifestyle diary. This study adds knowledge on the meaning of healthy eating, as well as on specific barriers to, and opportunities for, giving advice using an ECA in this target group. Furthermore, we translate this knowledge into general advice for designing an ECA in the context of health behaviour change, while reflecting on a co-design process with older adults.

Conceptualizing Usability for the eHealth Context: Content Analysis of Usability Problems of eHealth Applications.

BACKGROUND: Usability tests can be either formative (where the aim is to detect usability problems) or summative (where the aim is to benchmark usability). There are ample formative methods that consider user characteristics and contexts (ie, cognitive walkthroughs, interviews, and verbal protocols). This is especially valuable for eHealth applications, as health conditions can influence user-system interactions. However, most summative usability tests do not consider eHealth-specific factors that could potentially affect the usability of a system. One of the reasons for this is the lack of fine-grained frameworks or models of usability factors that are unique to the eHealth domain. OBJECTIVE: In this study, we aim to develop an ontology of usability problems, specifically for eHealth applications, with patients as primary end users. METHODS: We analyzed 8 data sets containing the results of 8 formative usability tests for eHealth applications. These data sets contained 400 usability problems that could be used for analysis. Both inductive and deductive coding were used to create an ontology from 6 data sets, and 2 data sets were used to validate the framework by assessing the intercoder agreement. RESULTS: We identified 8 main categories of usability factors, including basic system performance, task-technology fit, accessibility, interface design, navigation and structure, information and terminology, guidance and support, and satisfaction. These 8 categories contained a total of 21 factors: 14 general usability factors and 7 eHealth-specific factors. Cohen κ was calculated for 2 data sets on both the category and factor levels, and all Cohen κ values were between 0.62 and 0.67, which is acceptable. Descriptive analysis revealed that approximately 69.5% (278/400) of the usability problems can be considered as general usability factors and 30.5% (122/400) as eHealth-specific usability factors. CONCLUSIONS: Our ontology provides a detailed overview of the usability factors for eHealth applications. Current usability benchmarking instruments include only a subset of the factors that emerged from our study and are therefore not fully suited for summative evaluations of eHealth applications. Our findings support the development of new usability benchmarking tools for the eHealth domain.